# Multi-domain Pretraining

TinyLlama is a 1.1-billion-parameter small language model that adopts the same architecture and tokenizer as Llama 2, suitable for resource-constrained application scenarios.

A 13-billion-parameter language model for Russian, pretrained on 300GB of multi-domain data with Russian perplexity around 8.8

DistilBERT is a lightweight distilled version of BERT, retaining 97% of BERT's performance while being 40% smaller and 60% faster.

ProcBERT is a pre-trained language model specifically optimized for process texts, pre-trained on a large-scale corpus of process texts (including biomedical literature, chemical patents, and cooking recipes), demonstrating outstanding performance in downstream tasks.